Saving and retrieving locations of objects

ABSTRACT

Among other things, this document describes a computer-implemented method for storing and retrieving information about the locations of objects. The method can include receiving a first query that includes one or more terms identifying an object. The first query can be determined to include a command to store location information for the object. The first query can be parsed to determine identifying information for the object, and a location can be determined for the object. The method further includes identifying one or more attributes of the object that are not specified in the first query, and causing a first set of data to be stored that characterizes the identifying information for the objet, the location of the object, and the one or more attributes of the object.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.14/620,246, filed Feb. 12, 2015, which claims the benefit of U.S.Provisional Application No. 62/031,186, filed Jul. 31, 2014, theentirety of which is incorporated herein by reference.

TECHNICAL FIELD

This document generally relates to information storage and retrieval.

BACKGROUND

In the course of daily lives, people encounter many different objects.Objects routinely must be kept track of so that they can be found aftera period of non-use. Some objects are used frequently and routinely sothat a user can generally recall where the object is located inaccordance with the routine. For example, a user may charge her phone ona night stand each night, so that the phone can be expected to be on thenight stand the next morning. Similarly, a user may leave his car keysin a tray near the garage when he returns home from work each day, sothat the keys are easily located in the same spot the following morning.Sometimes, however, routine is broken and an object is left at anunusual location or lent to another person for a period of time. Otherobjects are used infrequently but must be readily accessible when needed(and some are even intentionally hidden), such as a hidden spare housekey, a birth certificate, or a vehicle title.

SUMMARY

This document generally describes techniques for saving and retrievinginformation about the locations of objects. In some implementations, auser can instruct an application or service on a computing device tostore information about the location of an object so that the locationcan easily be recalled at a later time upon request. For example, a usercan provide a query indicating that George is borrowing her rockcollection. In response to the query, an entry can be created in a datastructure for the user that captures the location of the rock collectionas being with George. Months later, the user may acquire a new stone forher collection, but cannot recall to whom she lent the collection orwhen. To facilitate her recollection, she may submit another query thatrequests the location of the rock collection, to which the applicationor service, based on the entry in the data structure, responds that therock collection is with George. The user can also indicate when thelocation of an object has changed, so as to cause location informationfor an object to be updated in the data structure. Thus, when Georgereturns the rock collection to the user, the return can be registered inthe data structure so that the current location of the object is nolonger represented as being with George. By maintaining the locations ofvarious objects in a data structure, sophisticated actions can beperformed in response to user queries. For example, notifications can betriggered when an object is moved from a particular location. In someimplementations, object location entries are stored in a data structurein a manner that permits relationships to be identified among one ormore objects and locations. For example, common objects or locationsnamed in a series of queries can be recognized as being the same objectsor locations that have previously been referenced so that informationabout the objects or locations can be reliably accessed or updated.Moreover, representations of objects or locations in the data structurecan be automatically associated with one or more attributes. The objectsor locations can then be identified by a user based on the attributes,even if the name of the object or location is not directly stated in aquery.

The techniques described herein may provide one or more of the followingadvantages. Using simple, natural language queries, a user can save,update, and retrieve information about object locations such asinformation that indicates where an object was last left, or informationthat identifies a person, organization, or other entity that is inpossession of the object. In some implementations, users can inputobject location queries by way of voice commands, which commands arethen processed based on particular sets of grammar that have beentrained on sample object location queries. The grammars can be used in amodel that probabilistically determines a particular type of voicecommand that a query corresponds to, without requiring the user to speakthe command using particular terms or particular language structure.Moreover, by associating stored objects and locations with one or moreattributes (i.e., concepts that are determined to be sufficientlyrelated to the objects and locations), users can subsequently refer tothe objects and locations indirectly based on their attributes, ratherthan directly based on the object or location names. Therefore,information about an object or location can be requested or updated withgreater flexibility than if strict recitation of the language that wasoriginally used to store object location information was required. Forexample, object location information may first be stored in response toa user query that “The rock collection is with George.” Later, the usermay want to know where the rock collection is, but uses only a shorthandreference to the rock collection, such as “rocks” or “stones.” Theshorthand reference of “rocks” may be sufficient to identify the “rockcollection” object by string comparison, and “stones” is a synonymassociated with the rock collection that is also determined to besufficiently closely related to the rock collection. Moreover,attributes can be determined about the rock collection and/or aboutGeorge, such as his full name and address. Using such attributes, theuser may inquire about any objects that are located at George's address,without naming George himself, for example. Likewise, a book may benamed by its author, a DVD movie by the starring actor or actress, and amusic album by its artist or genre, for example. Each of these examplesindirectly refer to an object rather than directly naming the objectitself. In addition, responses can be generated to more sophisticatedqueries by maintaining relationships among objects and locations in adata structure. For example, if object A was left at location X at afirst time, and object B was left at location X at a second time, thenlocation X can be associated with both objects, so that a user canobtain a response to a query that asks at once about all the objectsthat have been left at a particular location.

In some implementations, a computer-implemented method includesreceiving a first query that includes one or more terms identifying anobject. The first query can be determined to include a command to storelocation information for the object. The first query can be parsed todetermine identifying information for the object, and a location can bedetermined for the object. The method further includes identifying oneor more attributes of the object that are not specified in the firstquery, and causing a first set of data to be stored that characterizesthe identifying information for the objet, the location of the object,and the one or more attributes of the object.

These and other implementations may include one or more of the followingfeatures. The method can further include determining that the objectcorresponds to one or more entities in a data structure of entities,wherein the data structure includes representations of a plurality ofentities and maps relationships among particular ones of the pluralityof entities. Identifying the one or more attributes of the object thatare not specified in the first query can include selecting one or morefacts or categories that are associated with the one or more entities inthe data structure to which the object is determined to correspond.

The method can further include causing the first set of data for theobject to be stored in a personal objects data structure associated witha particular user account, wherein the personal objects data structure:(i) includes representations of one or more objects and one or morelocations, and (ii) maps relationships among particular ones of the oneor more objects and the one or more locations.

The data structure can map the relationships so as to indicate at leasta last known location for particular ones of the one or more objects.

The first query can further include one or more terms that identify thelocation of the object. The one or more terms that identify the locationof the object can include one or more terms that specify a geographicallocation of the object.

The first query can further include one or more terms that identify aperson or an organization with whom the object has been left. The methodcan further include associating the person or the organization with whomthe object has been left with a contact from an account of a user;detecting a user interaction with the contact that is associated withthe person or the organization with whom the object has been left; andin response to detecting the user interaction with the contact, causinga notification to be presented to the user pertaining to the object.

The method can further include determining that a user device is locatedwithin a threshold distance of the location of the object, and inresponse, causing a notification to be presented to a user pertaining tothe object.

The first query may not include information that identifies the locationof the object in content of the first query, and the location of theobject can be determined based on a current location of a computingdevice at which the first query was originally entered or spoken.

The method can further include receiving a second query that includesone or more terms that identify at least one of the one or moreattributes of the object. In response to receiving the second query, themethod can further include determining, based on the one or more termsthat identify the at least one of the one or more attributes of theobject, that the second query references the object that was identifiedin the first query; determining that the second query comprises acommand to perform an action associated with the object; and performingthe action associated with the object.

The second query can include a command to retrieve the location of theobject. Performing the action associated with the object can includeidentifying the location of the object from the first set of data andoutputting the location of the object.

The second query can include a command to update information about theobject. Performing the action associated with the object can includecausing updated information to be stored in the first set of data orother data.

The command to update information about the object can include anindication that the object is located at a new location. Causing theupdated information to be stored in the first set of data or the otherdata can include updating the stored location information for the objectto specify the new location of the object.

The first query can specify that a user has left the object at thelocation of the object. The second query can specify that the object hasbeen returned to the user. Performing the action associated with theobject in response to receiving the second query can include causinginformation to be stored that indicates that the object is no longerlocated at the location of the object identified in the first query.

The second query can indicate that the object is no longer located atthe location of the object identified in the first query. Performing theaction associated with the object in response to receiving the secondquery can include causing information to be stored that indicates thatthe object is no longer located at the location of the object identifiedin the first query. The method can further include, in response to adetermination that the object is no longer located at the location ofthe object identified in the first query, causing a notification to bepresented to a user that includes a reminder about one or more eventsassociated with the object.

The one or more terms in the first query that identify the location ofthe object can indicate that the object has been left with one or morefirst persons. The second query can indicate that the object has beenreturned from the one or more first persons, and the reminder about theone or more events associated with the object can include a reminder tomake the object available to one or more persons.

The method can further include identifying one or more attributes of thelocation of the object that are not specified in the first query. Thedetermination that the second query can include a command to perform theaction associated with the object is further based on the one or moreattributes of the location of the object.

The second query may not include any of the one or more terms thatidentified the object in the first query.

In some implementations, one or more computer-readable devices areprovided that have instructions stored thereon. When executed by one ormore processors, the instructions can cause performance of operationsthat include: receiving a query that includes one or more termsidentifying an object; determining that the query comprises a command tostore location information for the object; parsing the query todetermine identifying information for the object, and determining alocation of the object; identifying one or more attributes of the objectthat are not specified in the query; and causing a first set of data tobe stored that characterizes the identifying information for the object,the location of the object, and the one or more attributes of theobject.

In some implementations, a computer-implemented method can include:receiving, by a computing device, a first query that comprises a commandto store information specifying a location of an object; causing a setof data to be stored that identifies the object and the location of theobject; receiving, by the computing device, a second query thatcomprises a command to retrieve the location of the object, wherein thesecond query identifies the object by one or more terms that indicateone or more attributes of the object that were not specified in thefirst query; in response to receiving the second query: using the one ormore attributes of the object to identify the object from among aplurality of objects, and retrieving the location of the object from theset of data; and outputting information for presentation to a user thatidentifies the location of the object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an example process for saving andretrieving information about the locations of objects.

FIG. 2A is an example graphical representation of a data structure ofentities in which particular ones of the entities are interconnected.

FIG. 2B depicts an example portion of a data structure of interconnectedentities.

FIG. 3 shows a flowchart of an example process for saving and retrievinginformation about the locations of objects.

FIG. 4 shows an example of a system for managing and reportinginformation about the location of objects.

FIG. 5 shows an example of a computing device 500 and a mobile computingdevice that can be used to implement the techniques described herein.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 is a schematic diagram of an example process 100 for saving andretrieving information about the locations of objects. In someimplementations, the process 100 may be carried out by the system 400,depicted in FIG. 4 below. Generally, the process 100 permits a user touse unstructured natural language queries to store information thatindicates the locations of one or more objects. The process 100 canbuild a personal objects data structure that maps relationships amongobjects and their locations. The personal objects data structure can beused to respond to user queries that refer to objects and locationsindirectly or otherwise in different ways among successive queries. Forexample, based on particular paths through the personal object datastructure, one object may be identified in a query based on itsrelationship to another object; objects can be associated with alocation hierarchy (e.g., a shoe may be directly located in a closet,and the closet may be located in the guest bedroom of Sue's house);objects may be related by location; and object location histories can bedetermined. Moreover, objects and locations in the personal objects datastructure can be associated with attributes that represent conceptsrelated to the respective objects and locations. As described furtherbelow, the process 100 permits a user to identify an object or locationalong one or more dimensions, such as by direct use of the object orlocation name, partial or shorthand references to the object or locationname, and by indirect reference to one or more determined attributes ofthe object or location. Therefore, a user is not required to rememberthe particular terms that were originally used to describe an object andits location, but can instead later identify a previously stored objector location using a flexible vocabulary with a broader base of termsthat are descriptive of the object or location.

The process 100 begins at stage A (108), where a user 104 provides aquery 106 that includes an object-location storage command. The query106 states that the user 104 has “left the keys in the front pouch ofthe backpack.” Here, the query 106 is shown as a voice command utteredby the user 104, although the query 106 may also be entered by othermeans, such as by typing the text of the query 106 on a physical orvirtual keyboard. The query 106 can be input on a user computing device,such as a smartphone, tablet, netbook, wearable device (e.g.,smartwatch), or desktop computing device. The query 106 can be providedto an objects-and-location engine (“O&L engine”), that processesqueries, maintains a personal objects data structure 102 (e.g., adatabase, or another data repository), and performs one or moreoperations in response to queries. In this example, the process 100determines that the query 106 is an object-location storage commandusing a model that has been trained on various types of object-locationqueries for different types of object-location commands. Differentoperations can be performed in response to different command types. Eventhough the query 106 does not include a prefix or other carrier phrase(e.g., a hotword) that specifically corresponds to an object-locationstorage command (e.g., “Remember that I left the keys in the front pouchof the backback”), the model used by the O&L engine can determine fromthe context of the query 106 that it is most likely an object-locationstorage command, and not, for example, an objects information retrievalcommand. In some implementations, however, the O&L engine can beconfigured to permit or require command prefixes in the query thatidentify the type of command.

In response to receiving the query 106, the process 100 adds entries tothe user's personal objects data structure 102 to represent that theobject “keys” has been left at the location “front pouch of thebackpack.” An example representation of the personal objects datastructure 102 is shown in FIG. 1. The personal objects data structure102 is represented graphically here as a graph including nodes forrespective objects and locations, and edges that link the nodes andestablish relationships among the objects and locations. In someimplementations, the location named in the query 106 may be recorded inthe personal objects data structure in the manner the location wasidentified in the query 106. For example, the string “front pouch of thebackpack” may be stored as the location of the keys. In someimplementations, the named location may be separated into its componentparts, and separate, but related locations created for each part in thedata structure 102. Here, the location in the query 106 specifies alocation hierarchy. The “backpack” is a first location that may beidentified generically. However, the backpack also has sub-componentsincluding various compartments such as the “front pouch.” The “frontpouch” can thus be an individually identifiable element in the datastructure that is a sibling of the “backpack.” Therefore, the backpacknode 102 a is connected to front pouch node 102 b, which is in turnconnected to the keys node 102 c to indicate that the keys 102 c arecurrently located in the front pouch of the backpack. If the backpackwas previously represented in the data structure 102, additional orotherwise updated connections are made to the representation of thebackpack so as reflect the addition of the keys to the backpack. Thus,new queries can cause the O&L engine to make changes to and build uponthe data structure 102 if the data structure 102 previously existed. Ifa data structure 102 did not previously exist, one may be created.

At stage B (112), a second query 110 is provided by the user 104. Thesecond query indicates that the user's “Peyton Manning jersey is in thebackpack.” An additional entry can be made to the personal objects datastructure to reflect that an additional object is located in thebackpack. Thus, the jersey node 102 d is added and connected to thebackpack node 102 a.

At stage C (114), the process 100 identifies one or more attributes thatare determined to be relevant to the jersey object 102 d named in thequery 110. The attributes are associated with the object so that theymay be later used to identify the object in response to future userqueries that describe the jersey differently than how it was describedin query 110. In some implementations, the process 100 can tag an objector location with additional keywords for the determined attributes. Forexample, the representation of the jersey named in query 110 is taggedwith keywords 118 “quarterback,” “NFL,” “Colts,” “Broncos,”“Indianapolis,” and “Denver.” The keywords 118 can be determined fromone or more knowledge sources 116. In some implementations, theknowledge sources 116 may include or point to electronic resources suchas web pages and other electronic documents whose content discusses theobject or location that the query is referring to. In someimplementations, the knowledge sources 116 may include a curated datastructure that includes information about real-world entities includingvarious people, places, things, and ideas. Attributes (e.g., facts,topics, categories) for the entities can be organized in the datastructure. Therefore, a corresponding entity for an object or locationin a query can be identified, and the attributes in the externalentities data structure for the identified entity can be associated withthe object or location in the query. For example, the professionalquarterback, Peyton Manning, can be identified as a highly relevantentity for the “Peyton Manning jersey” object named in the query 110.The keywords 118 are attributes associated with the Peyton Manningentity, and therefore can be associated with the jersey object in thepersonal objects data structure 102 as well.

At stage D (120), the user 104 submits an object-location retrievalcommand in query 120 that asks, “Where did I leave my Colts jersey?”Although the user 104 would like to know the location of his PeytonManning jersey, as the object was previously described in query 110, thelater-submitted query 120 uses a different description of the sameobject: “Colts jersey.” Nonetheless, at stage E (124), the process 100correctly determines that query 120 is most likely referring to the“Peyton Manning jersey” that was earlier named in query 110 and that isrepresented in the personal objects data structure as node 102 d. Thecorrelation between the respective references to the “Colts jersey” andthe “Peyton Manning jersey” is determined based on the keywords for theattributes of the jersey that were identified at stage C (114). Theprocess 100 identified that Peyton Manning had a strong association withthe “Colts,” and therefore the term was tagged to the representation ofthe jersey 102 d. A match could then be determined between content inthe retrieval query 120 and the attributes of the jersey representation102 d.

At stage F (130), the user 104 submits a fourth query 128, “I have mykeys back.” This query 128 constitutes an object-completion type commandindicating that the keys have been returned. The process 100 canidentify from the context (e.g., structure and terms) of the query 128that the query 128 is an object-completion type command, and inresponse, update the personal objects data structure 102 at stage G(132) to reflect that the keys are no longer located in the front pouchof the backpack. For example, although not shown in FIG. 1, therepresentation of the keys in data structure 102 may be removed. In someimplementations, a current location of an object represented in the datastructure 102 may be expired in response to a completion-type commandrather than the object being deleted. In this way, the location historyof an object can be analyzed.

The change made at stage G (132) in response to query 128 is illustratedat stages H (136) and I (138). At stage H (136), the user 104 asks inquery 134, “What do I have in the backpack?” Because the personalobjects data structure 102 has been updated to reflect that the keys areno longer in the backpack, at stage I (138), the process 100 responds tothe fifth user query 134 with the answer, “Peyton Manning jersey,” whichis now the only object remaining in the backpack.

As noted above, among the one or more knowledge sources 116 that can beaccessed to identify attributes associated with an object or locationnamed in a query is an entities data structure. Examples of such a datastructure are depicted in FIGS. 2A and 2B.

FIG. 2A is an example graph 200 of an entities data structure inaccordance with an example implementation of the techniques describedherein. The data graph 200 may represent the storage and structure ofinformation in a data structure of interconnected entities. Such a datagraph 200 stores information related to nodes (entities) and edges(attributes or relationships), from which a graph, such as the graphillustrated in FIG. 2A can be generated. The nodes 202 may be referredto as entities, and the edges 204 may be referred to as attributes,which form connections between entities.

FIG. 2B depicts an example portion of a data structure 220 ofinterconnected entities. In some implementations, the data structure 220may store information about entities in the data structure in the formof triples. For example, triples identify a subject, property, and valuein the triples. Tom Hanks is an entity in the data structure 220 and isthe subject of the triples. A first of the triples 222 identifies theproperty ‘has profession,’ and a second of the triples 222 identifiesthe property ‘has spouse.’ The value of the property is the thirdcomponent of the triples. Tom Hanks has profession Actor′ and has spouse‘Rita Wilson.’ In the first of the triples 222, the property (attribute)has a value that is a fact in the data structure. Actor may or may notbe an entity in the data structure, for example. In someimplementations, the value component of the triplet may reference aclassification of the entity (e.g., Actor) from which a named-entityrecognition engine can determine a class for an entity mentioned in atext sample. In the second of the triples 222, the value component isanother entity in the data structure 220, particularly the entity forRita Wilson. Thus, the triple 222 specifies that the entity for TomHanks is connected or related to the entity for Rita Wilson by a spousalrelationship. Additional triples, and their conversely related triplesare shown in triples 224, 226, and 226′.

With reference to FIG. 3, a flowchart of an example process 300 is shownfor saving and retrieving information about the location of objects. Insome implementations, the process 300 can be carried out by any of thesystems described in this paper, such as the system 400 depicted in FIG.4. The process 300 can be performed on one or more devices. For example,the process 300 may be performed locally on a user device (e.g.,smartphone, tablet, notebook computer), on one or more servers, or maybe performed partially locally on a user device and partially remotelyon one or more servers.

The process 300 begins at stage 302 at which a user query is received.The query can be a structured query, a syntactic query, or a naturallanguage query, for example. The query may be typed or spoken by a user.For example, the query may be issued as a voice command that isprocessed by a speech recognizer and at least partially converted totext. On a user device, a user may access a voice assistant or othervoice application that accepts spoken input and that processes the inputto determine one or more actions to take in response. For example, thequery may be issued to a voice assistant that is configured to respondto various spoken commands (e.g., to send messages, to launchapplications, to save data, to set application parameters such as analarm clock, or to perform general web searches).

Among one or more commands that the voice assistant may be configured torespond to are commands for object location storage and retrieval. Atstage 304, the process 300 determines whether the received querycorresponds to one or more object location commands. If the query was avoice query, the utterance can be transcribed to text using an automaticspeech recognizer. The speech recognizer may include multiple grammarsthat correspond to different object location commands. In someimplementations, a carrier phrase in the query may specifically identifythe particular command. For example, in the query “Remember that Janehas my chemistry book,” the carrier phrase “Remember that” may indicatethat the query is a command to save the location of an object.

In some implementations, the speech recognizer may include multiplegrammars that each correspond to a respective type of object locationcommand. For example, the speech recognizer may include an assertionsgrammar, a completions grammar, a retrieval grammar, and a triggeractions grammar. Each of the grammars may be used to determine aparticular type of object location command that a query corresponds to.The grammars may be trained on text samples (e.g., natural languagequeries) that correspond to natural ways that users speak objectlocation commands in a language. In this way, for example, a carrierphrase may not be necessary for a voice assistant or other applicationto determine that a query is a particular type of object locationcommand. Instead, the naturally used words and structure of a query maybe sufficient to determine which type of object location command thatthe query corresponds to, even if no required carrier phrase or otherkeywords are included in the query. For example, the queries “Mychemistry book is with Jane,” “Jane has my chemistry book,” and “Thechemistry book is at Jane's” may all be recognized using assertiongrammars as being commands to store information about the location of anobject.

The assertion grammar is usable to determine that a query is a commandto store information about the location of an object. Some querypatterns that may be recognized with the assertion grammar as a commandto store objection location information include [$X is at $Y], [I lent$X to $Y], [I borrowed $X from $Y], [$X is in $Y], [$Y has $X], [I left$X with $Y], [Remember that $X is in the $Y]. Additional or other querypatterns may be understood by the assertion grammar for object locationstorage commands as well, which may be determined by statisticaltraining on object location storage command samples, for example.

The completion grammar is usable to determine that a query is a commandto register that an object has been returned or is otherwise no longerlocated at previously stored location. For example, if a user had madethe assertion that “I lent ‘James and the Giant Peach’ to Fred,” so asto cause the system to store information about the book ‘James and theGiant Peach’ being with Fred, then at a later time, an object completioncommand may be received from user input that “Fred returned ‘James andthe Giant Peach.’” The completion command indicates that the book hasbeen returned or is otherwise no longer located at a previously storedlocation, and the completion grammar is usable to recognize queries thatare completion commands. For example, the completion grammar mayrecognize query patterns for completion commands such as [$X returned$Y], [I got $X back], [Clear location of $X], [$X is no longer at $Y],etc. Additional or other query patterns may also be understood by thecompletion grammar, which may be determined by statistical training onobject completion command samples, for example.

The retrieval grammar is usable to determine that a query is a commandto identify the respective locations of one or more objects. Once thelocation of an object has been stored in response to an object locationstorage command, then at a later time, the user may submit an objectretrieval command to determine to recall where an object is located. Forexample, the location of a photo album that has previously beenregistered as being located in an attic cabinet can be recalled to theuser in response to the query “Where is the photo album?” The process300 can then identify that the photo album is located in the atticcabinet, and present to the user an indication that the photo album islocated in the attic cabinet. The response may be presented immediatelyin a textual message to the user such as an e-mail, SMS, or applicationtext. In some implementations, the response may be a spoken responsesuch as by speech synthesis. The retrieval grammar may recognize querypatterns for retrieval commands such as [Where is $X?], [Where are $X,$Y, and $Z?], [Where did I leave $X?], [Who has $X?], [Did I leave $Xsomewhere?], [Is $Y borrowing $X?], and others which may be determinedby statistical training on object location retrieval command samples,for example. In some implementations, the retrieval grammar mayrecognize retrieval commands that request identification of one or moreobjects that are stored at one or more locations. For example, a usermay ask “What are all of the items in the kitchen drawer?,” to which theprocess 300 may respond with the names of one or more items in thekitchen drawer.

The trigger actions grammar is usable to determine that a query is acommand to perform a specified action upon the occurrence of an event.The action and the event on which the action is conditioned may each bespecified by the user, or may be determined without being specified bythe user by using context or default settings. In some implementations,the event may be that an object has been left or returned from a givenlocation. In some implementations, the action may be to remind the userto do something with the object. For example, on January 5, a user mayrecord that he lent his “Harry Potter” collection to Sam. A week later,Jill tells the user that she would like to borrow the user's “HarryPotter” collection as soon as the collection is available. The user canspeak a voice query that commands that he be reminded to lend the “HarryPotter” collection to Jill when it is available. The command can berecognized using the trigger actions grammar. For example, the user maysay, “Remind me that Jill wants the ‘Harry Potter’ collection when Samis done with it.” Alternatively, the command may not include thecondition, but may instead be implied: “Remind me that Jill wants the‘Harry Potter’ collection.” From such commands, the process 300 cangenerate a reminder about providing the ‘Harry Potter’ collection toJill. Reminders may be periodically presented to the user, or the usermay ask what reminders he has set. In some implementations, the reminderis provided upon the occurrence of a completion event. For example, whenSam returns the object to the user, the user may speak a completioncommand, “I got the “Harry Potter” collection back.” In response toregistering the completion command, the process 300 may present anyrelevant reminders to the user about the object, such as “Reminder thatJill would like to borrow your ‘Harry Potter’ collection.” In someimplementations, the reminder can be provided upon the occurrence of anassertion (e.g., object location storage command). For example, the usermay set a first reminder in May that he must have his football padsready for practice by August 1. In June of that same year, the user maysubmit an object location storage command indicating that Bobby isborrowing the football pads. Upon issuing such command, the process 300may remind the user that the football pads must be returned by August 1in time for practice. The process 300 may also remind the user atparticular times in advance of the August 1 deadline that Bobby needs toreturn the football pads by August 1. Moreover, if the user makes aretrieval command, e.g., “Where are my football pads?,” the process 300can respond to the user with an indication of the location of thefootball pads (with Bobby), and also with a reminder that the footballpads are needed by practice on August 1. In some implementations, theuser can specify that a reminder about an object and/or its location beprovided to the user at particular times or recurrently.

In some implementations, the trigger actions grammar can be used to seta notification event that reminds a user that an object has been left ata particular location when the user interacts with a representation ofthe location. If a user indicates that an object has been left with aparticular person (or a particular organization or other entitygenerally), a contact corresponding to the person with whom the objectwas lent can be determined, such that when the user interacts with thecontact (e.g., sends an e-mail, an SMS message, a video call, socialmedia communication, phone call to the contact), a reminder can bepresented that the object is located with the person corresponding tothe determined contact. For example, if a user lends her snow shovel toPaula, the next time that the user calls Paula or that Paula calls theuser, a message can automatically be displayed to the user that remindsthe user that Paula has the snow shovel. In some implementations, ageographical location can be determined for the named location for anobject from a query, and a notification can automatically be generatedwhen the user (or the user's device) determines that its currentlocation is within a threshold distance of the determined geographicallocation. For example, a reminder message about the snow shovel can bepresented when the user drives through Paula's neighborhood so that theuser can conveniently pick up the object, for example, while the user isin Paula's area.

In some implementations, a query can include different combinations ofone or more objects, one or more locations, and one or more commands.For example, a compound query may be of the form [$A is at $X, and $B isat $Y], or [Where are $A and $B located?]. In such cases, the process300 can parse the query and perform similar actions for each of thecomponents of the query in a similar manner as is done with singlecomponent queries that are generally discussed in this paper forsimplicity. For example, a respective grammar may be used to recognizeeach component of a compound query, which grammars may be the same ordifferent depending on whether the individual components of the queryare directed to different commands. In some implementations, a singlecommand may be issued, but the process 300 can recognize multipleobjects and/or multiple locations associated with the command. Forexample, the object location storage command “My hat and scarf are inthe glove compartment” can be separated into the separate assertionsthat the hat is in the glove compartment and the scarf is in the glovecompartment, and such information may be stored as if separateassertions had been made for each of the hat and the scarf objects.

At stage 306, upon determining that the query includes a particular typeof object location command, the process 300 parses the query to identifythe object named in the query. One or more objects may be identifiedfrom the query. In some implementations, the name of the object may beextracted from the query and stored substantially in the manner that theobject was provided in the query. For example, given the object locationstorage command, “The Colts jersey is in the wash room,” the “Coltsjersey” string can be extracted and stored in a personal objects datastructure in the “wash room” location. In some implementations, one ormore related terms may be identified for the object that can be storedor used (depending on whether the query is a storage, completion, orretrieval command, for example) in place of or in addition to theoriginal object string extracted from the query. For example, synonymsor spell-corrected terms may be identified for an object. Thus, thephrase “Indianapolis Colts jersey” with the full team name may beidentified based on the partial name “Colts jersey” as used in thenatural language query. In some implementations, an extracted objectstring may be compared to other objects that have previously been storedand associated with locations in a personal objects data structure. Ifthe extracted object string or a related term is determined to besimilar or related to an existing (previously stored) object, then itmay be determined that the two objects are the same. If two objects arethe same, an additional entry in a personal objects data structure maybe created for the new instance of the object, which may be linked tothe previously stored object. In some implementations, if two objectsare determined to likely be the same object, then additional or updatedparameters may be generated for the existing representation of theobject stored in the personal objects data structure. For example, if awrench set was previously located in the tool shed, but a new queryindicates that the wrench set is now located in the garage, then theoriginal wrench set object representation may be accessed, and itscurrent location updated from the tool shed to the garage. In someimplementations, an object string may be extracted from the query and anentry stored for the object without performing a comparison of theextracted object string to other objects at the time of extraction andstorage. At a later time, when a retrieval or completion command isreceived from a user, for example, the process 300 may identify allentries that match the retrieval or completion command, includingidentifying common objects. For example, the user may assert that his‘cards’ are in the office at a first time, and then at a later timeassert that his ‘deck of cards’ is in the basement. The determination asto whether the ‘cards’ object and the ‘deck of cards’ object are thesame may be performed when the ‘deck of cards’ assertion is made, or maybe performed when a user later asks, “Where are the last two places thatI left my cards?” If the process 300 determines that the partial objectstring ‘cards’ identifies the same object as the ‘deck of cards,’ thenthe process 300 may respond to the last query with an indication thatthe last two locations for the cards were the basement and the office.

At stage 308, the process 300 identifies the location of the object fromthe query. The location may be identified using similar techniques tothose used to identify the object from the query. For example, the querymay be parsed, and one or more words extracted that correspond to thelocation for the object. For an object location storage command, the oneor more words that correspond to the location may be saved in thepersonal objects data structure and associated with the object. In someimplementations, the object location specified in a new query may bedetermined to be the same as other locations in the personal objectsdata structure that have been previously recorded. In someimplementations, a relationship may be determined among multiplelocations identified in a particular query and/or among locations in aquery and locations that have been previously recorded in the personalobjects data structure. For example, a previously saved location in thedata structure may be “office.” A new query may indicate that an objectis located in the “office closet.” A hierarchical relationship may beestablished between the office and the office closet so that the objectin the office closet may also be identified as being in the office,although not all office objects may be in the office closet. In someimplementations, geographical location information may be associatedwith an object rather than a conceptual location. For example, ageographic location may be implied from some queries that do notexplicitly recite a location, or from queries that do recite a locationfor which a geographic location can be identified. Thus, the query“Remember that I left my car here,” may cause the process 300 to uselocating means such as GPS to determine the geographic coordinates ofthe device that received the query, and such coordinates can be recordedas the current location of the car. In another example, a user might saythat “I left my coat at Jerry's World Famous Deli.” The geographiclocation of Jerry's World Famous Deli may be determined, and may bestored rather than or in addition to the name of the deli in thepersonal objects data structure.

At stage 310, the process 300 determines that the object identified froma query corresponds to one or more entities. Entities can includepersons, places, things, or ideas, for example. Examples of entities arewide ranging and may include anyone and anything from President BarackObama and the Constitution of the United States of America to Jon BonJovi, reality shows and kangaroos, to books, movies, businesses, andmore. The one or more known entities may have been previouslyidentified, or may be identified after a query is received. In someimplementations, where the entities have been previously identified, theentities may be indexed in a data structure of entities. The datastructure may include one or more sub-data structures and may berepresented as a graph, in some implementations (see e.g., FIG. 2A). Forexample, each entity may be represented by a node in the graph,particular ones of the entities may be connected to each other in thegraph by way of one or more relationships (e.g., the entity representingJ. K. Rowling may be connected to the entity representing the book‘Harry Potter and the Philosopher's Stone’ through a bi-directionalrelationship that indicates that J. K. Rowling authored the book), eachof the entities may be associated with a respective set of attributes,and the entities may be associated with one or more topics (categories).For example, the entity representing J. K. Rowling may be connected toother entities such as the entity representing Yate, United Kingdom asthe author's place of birth. The J. K. Rowling entity may also haveattributes associated with her such as date of birth, interests, etc.Information about entities may be hand-seeded by users and/or may bescraped from publicly available information (e.g., web pages, blogs, andother electronic documents) and relied upon as accurate data about anentity if at least a threshold level of confidence is determined basedon the information source (e.g., based on the reliability of the sourceof the data, the frequency that the data is repeated across multiplesources, and the staleness of the data). The entities data structure maybe represented, in some implementations, in a manner similar to theentities graph 202 depicted in FIGS. 2A and 2B. In some implementations,information about an entity that corresponds to an object or a locationin a query can be identified after receipt of the query, such as byperforming an internet search based on the query in order to identifypublicly available information that is relevant to the query.

At stage 312, one or more attributes are determined about the entitiesthat have been determined to correspond to the object specified in thequery. The attributes of the entity are generally usable to identifyadditional information that is relevant to the object but that wasunspoken in the query. The stored object can then be tagged or otherwiseassociated with the attributes of its corresponding entities so thatusers can subsequently identify the object by reference to theattributes. In some implementations, additional keywords are associatedwith the object based on the attributes. For example, in response to aquery indicating that a user has left her copy of “To Kill aMockingbird” with Ryan, data is caused to be stored in the personalobjects data structure representing that the object “To Kill aMockingbird” is currently at the location with “Ryan.” The object “ToKill a Mockingbird” may be determined at stage 310 to correspond to theclassic novel by Harper Lee (i.e., the novel is the entity to which thequery object corresponds in this example). Since the object referencedin the query is determined to likely correspond to the novel, additionalattributes about the novel can be associated with the object. Thus, theauthor's name (Harper Lee), original publication year (1960), themes(racial injustice, law), genre (Southern Gothic, fiction), and majorcharacters (Atticus Finch) are all attributes that describe the book.Keywords can be identified from the attributes and associated with thestored object, such as “book,” “Harper Lee,” 1960, “racial injustice,”“law,” “Southern Gothic,” and “Atticus Finch.” Thus, even though theuser originally indicated that “To Kill a Mockingbird” was with Ryan,different terms can be used to later reference the object. For example,the user may issue a retrieval command, “Who has my book about AtticusFinch?” or a completion command, “The book by Harper Lee has beenreturned,” and the process 300 can then identify that the object inthese subsequent commands is the “To Kill a Mockingbird” book that Ryanhad borrowed. Similarly, synonyms, short-hand object names, full-lengthobject names, categories, and topic or category information can betagged with an object so that the object can be referenced usingalternative terms from the terms that were previously used to saveinformation about the object.

In some implementations, entity attributes can be identified byaccessing an entities data structure that includes information aboutentities and their relationships. Sometimes, multiple entities may bedetermined to correspond to an object in a query. For example, “To Killa Mockingbird” might not refer to the book, but instead to the 1962 filmadaptation of the book starring Gregory Peck. Where the object specifiedin the query is ambiguous, and may correspond to one of multipledifferent candidate entities, then attributes from one or more of thecandidate entities may be associated with the object.

To facilitate selection of candidate entities, confidence scores can bedetermined for the candidate entities that reflect respectivelikelihoods that each candidate entity matches the object indicated in auser query. The confidence score can be used to select which entities'attributes to associate with an object. In some implementations, theattributes of the n entities having the n highest confidence scores canbe associated with an object. In some implementations, only entitieswhose confidence scores meet (e.g., exceed) a threshold confidence scoremay have its attributes associated with an object. The confidence scorefor an entity may be based on one or more factors including the contextof the query that named the object, and the frequencies with whichrespective entities are found in one or more corpora of electronicresources. For example, the book “To Kill a Mockingbird” may bediscussed ten times more frequently than the movie on internet pages orother resources that include the phrase “To Kill a Mockingbird.”Therefore, the entities data structure may include informationindicating the relative popularity of the book as compared to the movie,and such information may result in the confidence score for the bookbeing higher than the confidence score for the movie. Similarly, if thequery itself included context that disambiguated candidate entities,such context could be used to influence the confidence scores of thecandidate entities. For example, it is clear from the query, “The DVDsfor To Kill a Mockingbird′ are at Ryan's place,” includes a reference tothe movie, rather than the book. The confidence score for the movieentity could therefore be made higher than the confidence score for thebook in this example, and the attributes for movie could be associatedwith the object specified in the query, rather than the attributes forthe book.

At stage 314, one or more additional attributes that are not associatedwith a corresponding entity can be associated with the query object. Theadditional attributes can include terms, phrases, and other informationthat relate to the object but that are not necessarily identified froman entity for the object. Even where a corresponding entity cannot beidentified from an entities data structure for an object, otherinformation sources may indicate relevant attributes for the object. Forexample, a personal object that has a particular, and possibly private,meaning may not be an entity represented in an entities data structure(which may be generated based largely on publicly availableinformation). Thus, “Grandma's hand-knit sweater” may be an objectwithout an entity in the entities data structure. Still, keywordattributes such as “Grandma,” “family,” “heirloom,” “sewn,” and “shirt”may be associated with the object to permit expanded retrievalcapabilities when the object is later referenced in a query by the user.The additional attributes may be identified in some implementations byaccessing a thesaurus to determine keyword synonyms, by identifyingrelated (but not corresponding) entities from an entities datastructure, and/or by performing a web-based search for the object andidentifying keywords from relevant search results.

Additional attributes for an object can also be determined based on useraccount data. For example, an object may be described in data stored inan account associated with the user, which description can be parsed toidentify keyword attributes to associate with the object. This can bebeneficial, for example, to determine attribute information for objectsregardless of whether an entity corresponding to the object isrepresented in an existing entities data structure. Thus, keywords orother data derived from the description of an object from a user'se-mails, applications, social media accounts, profile information, orother sources can be stored in association with an object in a personalobjects data structure. A user may opt-in or opt-out of having useraccount data analyzed for such purposes, and may restrict what data isanalyzed.

At stage 316, information identifying the object specified in the query,the location of the object, and any attributes associated with theobject are stored in a personal objects data structure. The personalobjects data structure may include a collection of data structures inone or more locations so that the data, although associated, may or maynot be stored together. Information may be stored at stage 316 of theprocess 300 when the query is a command to store information about thelocation of an object. Queries for other commands, such as commandsrequesting information about the location of an object, may not causenew information to be stored about an object or its location. Ininstances where object location information is stored, additionalinformation may also be stored in addition to the object identifier,location, and object attributes. For example, keywords or otherattributes for locations may be identified using similar techniques tothose used for object attributes (e.g., by determining the attributes ofan entity corresponding to the location named in a command). Locationattribute information can therefore be stored with an object-locationentry. A timestamp that marks the date and time of when an object wasleft at a particular location or when a query was received can also bestored with an object-location entry. The timestamp can be used toperform sophisticated object-location lookup operations. For example,the timestamp can be used to respond to user queries that ask when anobject was left at a location, for how long an object has been at alocation, or where all the locations are that an object has been locatedat within a particular period of time.

In some implementations, information stored in the personal objectsrepository can be represented as a graph. The graph may comprise nodes(vertices) representing the various objects and locations stored in therepository, and edges that connect the nodes which representrelationships among the nodes. The graph may be represented in a similarmanner as the graph shown depicted in FIG. 2. For example, a particularlocation, at which one or more objects are or have been located, may berepresented by a node connected to other nodes for each of the one ormore objects. Objects that have been located at one or more locationsmay have nodes that are respectively connected to other nodes for theirrespective locations. In some implementations, attributes for objects orlocations can also be represented by nodes in the graph. For example,the object “To Kill a Mockingbird” may correspond to the book by HarperLee, which may have been determined by accessing an entities datastructure and matching the object to the book represented as an entityin the data structure. The author of the book, Harper Lee, may then bean attribute associated with the object and may be represented in thepersonal objects graph as a node connected to the “To Kill aMockingbird” object node.

At stage 318, a second query is received. The second query may refer toan object that has previously been stored in the personal objectsrepository. For example, the first query received at stage 302 may havebeen an object-location storage command, and the second query may be aretrieval command that requests the process 300 to respond with thelocation of the object from the first query. The second query may usethe same, similar, or even very different language from the first queryto refer to the same object. For example, the first query may refer to anovel by its title (e.g., “To Kill a Mockingbird”), whereas the secondquery may refer to the novel by only a portion of its title (e.g.,“Mockingbird”), or even by a completely different parameter such as bythe author (e.g., “Harper Lee”) or another attribute of the novel.Similarly a particular location specified in the query may be stated thesame, similarly, or very differently from how the same location wasspecified in an earlier query. For example, one query may identify thatan object is located at “Mom's place,” whereas the second queryidentifies the same location as a street address for Mom's place, or atcorresponding geographic coordinates for Mom's place.

At stage 320, the second query is parsed. The process 300 determineswhether the query is a type of object-location command, and if so, whichtype of command (e.g., object-location storage command (assertion),completion command, retrieval command, trigger actions command). Thevarious assertion, completion, retrieval, trigger action, and othergrammars can be used to parse a spoken query, for example. The objectand/or location string in the query is identified by parsing the query.For example, the query “Where did I leave the Harper Lee book?” can beparsed and determined to be a retrieval type command having objectstring “Harper Lee book” and no location specified.

At stage 322, the process 300 determines that an object and/or locationspecified in the second query is the same as an object and/or locationspecified in the first query that was previously received. If anidentified object and/or location string from the second query matches apreviously received object and/or location string from the first query,then the object and/or locations may be determined to be the samebetween the queries. If the same object or location is identifieddifferently in the second query than how it was identified in the firstquery, then one or more of the closest matching objects or locations,respectively, can be identified. In some implementations, a stringcomparison is performed between portions of the second query andportions of the first query such as a comparison between the objectstring in the same query and the object string extracted from the firstquery. In some implementations, the object named in the second queryindirectly references the same object named in the first query by one ormore attributes of the object. For example, the “Harper Lee book” is anindirect reference to “To Kill a Mockingbird.” If the second queryincludes the object string “Harper Lee book,” then such string may bematched to “To Kill a Mockingbird” by comparing the object string“Harper Lee book” to the attributes for the book titled “To Kill aMockingbird,” and a match can be determined from the book's author. Insome implementations, the attributes are keywords which have been taggedto the “To Kill a Mockingbird” object that was previously stored in thepersonal objects data structure, and a match is determined by performinga comparison of an indirect object specified in the second query to thekeywords of previously stored objects. In this manner, the process 300can identify one or more objects in a personal objects repository thatlikely correspond to an indirectly referenced object in a query. In someimplementations, entries for objects or locations in a personal objectsdata structure can be updated when corresponding objects or locationsare referenced in a series of queries, regardless of whether thereferences to the objects or locations are the same or different. Forexample, an entry for a location named “Bill's Apartment” can be createdin the data structure in response to receiving a first query thatindicates a first object is located at Bill's apartment. A second querymay indicate that a second object is located at Bill's apartment, butmay identify Bill's apartment by the street address 555 Candy Cane Lane,Apartment No. 3. The process 300 can determine that the address statedin the second query corresponds to Bill's apartment, and therefore, thelocation entry for Bill's apartment is updated to indicate that thesecond object is also located at Bill's apartment. For example, in thegraph representation of the personal data structure a node of the secondobject can be branched off of the node representing the location for“Bill's apartment.”

At stage 324, the process 300 performs an action associated with theobject responsive to the second query. For example, identificationinformation for an object and its current location can be saved forlater retrieval; information about objects or locations can be retrievedand provided to a user; completion actions can be performed; and triggeractions can be performed in response to the second query.

FIG. 4 depicts an example system 400 for managing and reporting locationinformation for objects. The system 400 can be configured to receiveuser queries to store information about where a user has left personalobjects so that the locations of such objects can later be recalled atthe request of the user. The system 400 permits a flexible querystructure and natural language to be used in queries to command thesystem 400 to store information about objects and locations of theobjects, to perform operations on stored representations of the objectsand their locations, and to respond to queries with information that isusable to present saved information about objects and locations tousers. The system 400 can include one or more of a user device 402, anetwork 410, an objects and locations engine 404, an entities datastructure 408, and a speech recognizer 406. Although various componentsof the system 400 are depicted as being separate from each other, insome implementations, particular ones of the components can beintegrated with other components in various combinations, and some ofthe components may be separated into additional sub-components. Forexample, the speech recognizer 406 may be implemented on the user device402 or on one or more server systems remote from the user device 402.Similarly, the objects and location engine 404 may be provided locallyon the user device 402, on one or more server systems remote from theuser device 402, or particular sub-components may be provided locallyand others provided on remote servers. In some implementations, thesystem 400 can be configured to perform particular processes that aredescribed in this paper, such as process 100 depicted in FIG. 1 orprocess 300 depicted in FIG. 3.

The user device 402 is configured to receive queries from a user and topresent information to the user. The user device 402 may be asmartphone, tablet computing device, desktop computer, notebookcomputer, or smartwatch in some implementations. The user device 402 maybe configured to receive queries based on spoken input from a user orbased on textual input typed by the user. The user device 402 may alsoinclude or be operatively coupled to an electronic display, such as atouch-sensitive display, that can present information to a user. Theuser device 402 can include I/O interface 412, speech processing engine414, display manager 416, time and location engines 418, and an accountmanager 420. The I/O interface 412 is configured to send and receivedata from the user device 402. For example, the I/O interface 412 mayinclude one or more wireless communication chips that can send andreceive data over network 410, such as the internet or a privatenetwork. The speech processing engine 414 can perform speech recognitionon spoken input from a user. In some implementations, the speechprocessing engine 414 may have a limited grammar that only recognizes alimited set of voice commands. The audio for spoken input of complexnatural language queries can be transmitted, in some implementations, tothe speech recognizer 406 for remote processing on servers having moresophisticated speech recognition capabilities and larger grammars thanthose on the user device 402. In some implementations, all speechprocessing is performed on the user device 402. The display manager 416manages what is displayed on one or more electronic displays associatedwith the user device 402. Queries may be presented to the user as thequery is received by the device 402, and query responses may be shown onthe electronic display for presentation of the response to the user.

The time and location engines 418 are configured to track the currenttime and location of the user device 402. Using information from thetime and location engines 418, the system 400 can associate user querieswith a time and/or location. A timestamp can be associated with a query,for example, to indicate the date and time at which an object was leftat a particular location. Thus, when the user makes an object-locationstorage command, such as “I lent the basketball to Paige thisafternoon,” the timestamp is associated with the command (query) inorder to mark the date and time that the basketball was lent to Paige.In some implementations, a user can expressly specify when an object wasleft at a location, and such express timestamp supersedes a default,current timestamp from the time and location engines 418. For example, aquery received on June 5 that stated, “I gave the basketball to Paige 4days ago,” can be tagged with the date June 1, rather than June 5, sincethe content of the query expressly indicates a date. By recording thedate and time when an objects were left at particular locations, thesystem 400 can later use such information to respond to temporal-basedqueries regarding objects and locations. For example, the system 400 mayuse a timestamp associated with entries in a personal objects datastructure to respond to queries about when an object was left at one ormore locations, how long an object has been left at one or morelocations, and where all the locations are that an object has been leftat within a particular period of time.

The account manager 420 of the user device 402 manages one or more useraccounts on the device 402. For example, the account manager 420 canmanage respective accounts for each of multiple users. Each user accountmay include user profile information as well as information about userpreferences related to object-location type commands. User accountinformation can be stored in the user profile repository 422. One ormore personal objects data structures for respective user accounts canbe stored in an objects data repository 424. Therefore, different userscan have their own, personal data structure separate from that of otherothers. When a first user is logged into the device 402 in a firstaccount, the he may save and retrieve information about the locations ofobjects in his respective personal objects data structure. When anotheruser is logged into the device 402 in a second account, that user maysave and retrieve information about the locations of objects in hisrespective personal objects data structure. The accounts may beassociated with the user device 402 (e.g., maintained by an operatingsystem of the user device 402), or by one or more services orapplications on or otherwise accessible to the device 402. For example,an objects-location application installed on the device may permitmultiple different users to log into the application and may maintainrespective accounts for each of the users. In some implementations, atleast one of the user profile repository 422 and the objects datarepository 424 can be maintained at least partially remotely from theuser device 402, such as at an objects-locations server. In someimplementations, user data, including the personal objects datastructure, may be maintained both locally and remotely. Changes to theuser data, such as updates to the personal objects data structure, canbe automatically propagated between one or more local and remote deviceswhen a change is made at one of the devices.

The objects and locations engine 404 (“O&L engine”) is configured toreceive, process, and respond to queries including one or more types ofobject-location commands. The O&L engine 404 can receive a query, parsethe query, and respond based on the content of the query. For example,when a user issues a voice query at the user device 402, the user device402 may send the query or a characterization of the query over network410 to the O&L engine 404, which may be hosted on one or more remoteservers. The O&L engine 404 can perform one or more actions in responseto the query, such as storing information about where an object islocated, identifying where an object is located, updating informationabout where an object is located, determining information for a triggeraction, and generating instructions to send to the user device 402 tocause performance of an action in response to a trigger event.

The O&L engine 404 can include an I/O interface 426, a query parser 428,a named entity recognition engine 430, an O&L data manager 432, and anaccounts manager 434. Moreover, the O&L engine 404 can communicate witha speech recognizer 406 and one or more external data sources such as anentities data structure 408. The I/O interface 426 is configured tocommunicate over one or more networks 410, such as a private network orthe internet, and can communicate with the speech recognizer 406, theentities data structure 408 and other external sources, and with theuser device 402.

The query parser 428 is configured to parse queries received from theuser device 402. The query parser 428 can process a user query todetermine whether it is an object-location command, and if so, theparticular type of object-location command (e.g., object-locationstorage command, retrieval command, completion command, trigger actionscommand, or another type). The query parser can identify each objectnamed a query, each location named in a query, and a correspondencebetween objects and locations named in a query. For example, a query mayindicate that “The towels are in the dryer and the sheets are in thecloset.” The query parser 428 can identify that the query is anobject-location storage command that includes references to two objects(towels and sheets), two locations (dryer and closet), and that thetowels and dryer correspond as an object-location pair, and the sheetsand closet correspond as another object-location pair.

In some implementations, the query parser 428 uses one or more grammarsto process and parse a query. For example, a voice query may betranscribed by the speech recognizer 406. The speech recognizer 406includes grammars that are configured to recognize various types ofnatural language object-location commands. The assertions grammar 440 isconfigured to permit recognition of object-location storage commands(e.g., “The hockey puck is in the backseat,” “Remember that the artcollection is at Bill's place.”). The retrieval grammar 442 isconfigured to permit recognition of object retrieval commands (e.g.,“Where is the hockey puck?”, “What does Bill have?”). The completiongrammar 444 is configured to permit recognition of objection completioncommands (e.g., “I've returned the hockey puck,” “Bill no longer has theart collection.”). The event trigger grammars 446 are configured topermit recognition of event trigger commands (“Remind me that Sallywants the hockey puck,” “Remind me that Sally wants the art collectionafter Bill.”). The query parser 428 can send a query to be transcribedby the speech recognizer 406 with the various grammars 440-446 of thespeech recognizer 406. From the transcription of the voice query, thequery parser 428 can extract one or more character strings correspondingto objects and locations in the strings. In some implementations, thegrammars 440-446 can be used to identify which terms in a textual querycorrespond to objects and locations, and also to determine theparticular type of object-location command that a query is. For example,each grammar may be included in one or more language models that havebeen trained on a large number of training queries of a particular type.The assertion grammars 440 can be based on training samples ofassertion-type queries (e.g., object-location storage commands), theretrieval grammars 442 can be based on training samples ofretrieval-type queries, the completion grammars 444 can be based ontraining samples of completion-type queries, and event trigger grammars446 can be based on training samples of event trigger-type queries. Whena new query is received, a model that has been statistically trained onqueries of the various types can make a probabilistic determination ofthe particular type of query that has been made (e.g., assertion,retrieval, completion, event trigger, other).

The named entity recognition engine 430 is configured to determine oneor more attributes of an object or location named in a query. The namedentity recognition engine 430 may determine attributes for eitherobjects or locations in a query, or for both objects and locations. Ifmore than one object or location is provided in a query, the engine 430can determine attributes for all or only some of the objects orlocations. The named entity recognition engine 430 can access one ormore data sources to identify the attributes for an object or location.In some implementations, the data sources may be external to the system400. For example, the named entity recognition engine 430 may perform asearch of one or more corpora based on the query or based on the objector location of interest. The content of the search results can then beused to determine one or more keywords associated with the object orlocation. For instance, the named entity recognition engine 430 mayidentify keyword attributes for the object “camera” by the performing asearch for the word camera, and determining the most relevant keywordsassociated with camera from one or more of the top-ranked searchresults. These keyword attributes can then be associated with the cameraobject so that the camera can later be identified based on itsattributes (e.g., photograph, lens, viewfinder, D-SLR, etc.) rather thanor in addition to the original manner in which the camera was identifiedin a query.

In some implementations, the named entity recognition engine 430determines attributes for an object or location by identifying one ormore entities that the object or location corresponds to in the realworld, and associating attributes for those entities with the object orlocation named in the query. In one example, the entity and itsattributes are identified with reference to an entities data structure408. The entities data structure 408 generally includes a structuredrepresentation of entities (e.g., persons, places, things, ideas),relationships among entities, and attributes of such entities. Theentities data structure 408 may be represented as a graph having nodesrepresenting entities and edges connecting the nodes representingrelationships among entities and various attributes (e.g., facts) aboutthe entities. One example of such a graph is shown in FIGS. 2A and 2B.The entities data repository 448 includes data for the entitiesrepresented in the data structure 408, and the attributes datarepository 450 includes data regarding the attributes of the entities.The entities data structure 448 may be procured manually, automatically,or a combination of both. For example, entities may be determined bycrawling publicly available resources on the internet, analyzing thecontent of the publicly available resources, and making a determinationabout the existence and attributes of entities based on the analysis ofthe resources. For instance, the topic of a large number of resourcesmay be a particular object (or series of objects), such as the SherlockHolmes series of books by Arthur Conan Doyle. A high confidence may bedetermined that each book in the series is an entity, as is theirauthor, Sir Arthur Conan Doyle. Moreover, various attributes may bedetermined for the books such as a genre (mystery), and the time andplace that they were written and first published. Such information maybe stored in the entities data structure 448.

In some implementations, the named entity recognition engine 430 mayassociate an object or location in a query with an entity in theentities data structure 448, such as by a pointer that references theassociated entity. This way, if information is added or otherwiseupdated about an entity, such information can be updated for thecorresponding object or location from the query as well.

The O&L data manager 432 manages information about objects andlocations. For example, working with the accounts manager 434 and inresponse to user queries, the O&L data manager 432 can store and accessinformation about objects and object storage locations for users. Once aquery has been parsed and object or location attributes determined, theO&L data manager 432 can cause such information to be stored in a by thequery parser 428 and named entity recognition engine 430, respectively,information regarding an object, its location, and attributes of theobject and location can be stored in a personal objects data structure438, which is maintained by the accounts manager 434. For example, asindicated by data in the user accounts repository 436, a respectivepersonal objects data structure 438 can be maintained for each ofmultiple user accounts. Therefore, for example, Fred can save andretrieve information about where his own personal objects are located,and Emma can do the same. The O&L data manager 432 can also pullinformation from the personal objects data structure 438 to respond toqueries, such as object retrieval queries, for example.

FIG. 5 shows an example of a computing device 500 and a mobile computingdevice that can be used to implement the techniques described herein.The computing device 500 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The mobile computing device is intended torepresent various forms of mobile devices, such as personal digitalassistants, cellular telephones, smart-phones, and other similarcomputing devices. The components shown here, their connections andrelationships, and their functions, are meant to be examples only, andare not meant to limit implementations of the subject matter describedand/or claimed in this document.

The computing device 500 includes a processor 502, a memory 504, astorage device 506, a high-speed interface 508 connecting to the memory504 and multiple high-speed expansion ports 510, and a low-speedinterface 512 connecting to a low-speed expansion port 514 and thestorage device 506. Each of the processor 502, the memory 504, thestorage device 506, the high-speed interface 508, the high-speedexpansion ports 510, and the low-speed interface 512, are interconnectedusing various busses, and may be mounted on a common motherboard or inother manners as appropriate. The processor 502 can process instructionsfor execution within the computing device 500, including instructionsstored in the memory 504 or on the storage device 506 to displaygraphical information for a GUI on an external input/output device, suchas a display 516 coupled to the high-speed interface 508. In otherimplementations, multiple processors and/or multiple buses may be used,as appropriate, along with multiple memories and types of memory. Also,multiple computing devices may be connected, with each device providingportions of the necessary operations (e.g., as a server bank, a group ofblade servers, or a multi-processor system).

The memory 504 stores information within the computing device 500. Insome implementations, the memory 504 is a volatile memory unit or units.In some implementations, the memory 504 is a non-volatile memory unit orunits. The memory 504 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 506 is capable of providing mass storage for thecomputing device 500. In some implementations, the storage device 506may be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The computer program product can also be tangiblyembodied in a computer- or machine-readable medium, such as the memory504, the storage device 506, or memory on the processor 502.

The high-speed interface 508 manages bandwidth-intensive operations forthe computing device 500, while the low-speed interface 512 manageslower bandwidth-intensive operations. Such allocation of functions is byway of example only. In some implementations, the high-speed interface508 is coupled to the memory 504, the display 516 (e.g., through agraphics processor or accelerator), and to the high-speed expansionports 510, which may accept various expansion cards (not shown). In theimplementation, the low-speed interface 512 is coupled to the storagedevice 506 and the low-speed expansion port 514. The low-speed expansionport 514, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 500 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 520, or multiple times in a group of such servers. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 522. It may also be implemented as part of a rack server system524. Alternatively, components from the computing device 500 may becombined with other components in a mobile device (not shown), such as amobile computing device 550. Each of such devices may contain one ormore of the computing device 500 and the mobile computing device 550,and an entire system may be made up of multiple computing devicescommunicating with each other.

The mobile computing device 550 includes a processor 552, a memory 564,an input/output device such as a display 554, a communication interface566, and a transceiver 568, among other components. The mobile computingdevice 550 may also be provided with a storage device, such as amicro-drive or other device, to provide additional storage. Each of theprocessor 552, the memory 564, the display 554, the communicationinterface 566, and the transceiver 568, are interconnected using variousbuses, and several of the components may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 552 can execute instructions within the mobile computingdevice 550, including instructions stored in the memory 564. Theprocessor 552 may be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor 552may provide, for example, for coordination of the other components ofthe mobile computing device 550, such as control of user interfaces,applications run by the mobile computing device 550, and wirelesscommunication by the mobile computing device 550.

The processor 552 may communicate with a user through a controlinterface 558 and a display interface 556 coupled to the display 554.The display 554 may be, for example, a TFT (Thin-Film-Transistor LiquidCrystal Display) display or an OLED (Organic Light Emitting Diode)display, or other appropriate display technology. The display interface556 may comprise appropriate circuitry for driving the display 554 topresent graphical and other information to a user. The control interface558 may receive commands from a user and convert them for submission tothe processor 552. In addition, an external interface 562 may providecommunication with the processor 552, so as to enable near areacommunication of the mobile computing device 550 with other devices. Theexternal interface 562 may provide, for example, for wired communicationin some implementations, or for wireless communication in otherimplementations, and multiple interfaces may also be used.

The memory 564 stores information within the mobile computing device550. The memory 564 can be implemented as one or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. An expansion memory 574 may also beprovided and connected to the mobile computing device 550 through anexpansion interface 572, which may include, for example, a SIMM (SingleIn Line Memory Module) card interface. The expansion memory 574 mayprovide extra storage space for the mobile computing device 550, or mayalso store applications or other information for the mobile computingdevice 550. Specifically, the expansion memory 574 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, theexpansion memory 574 may be provide as a security module for the mobilecomputing device 550, and may be programmed with instructions thatpermit secure use of the mobile computing device 550. In addition,secure applications may be provided via the SIMM cards, along withadditional information, such as placing identifying information on theSIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory(non-volatile random access memory), as discussed below. The computerprogram product contains instructions that, when executed, perform oneor more methods, such as those described above. The computer programproduct can be a computer- or machine-readable medium, such as thememory 564, the expansion memory 574, or memory on the processor 552. Insome implementations, the computer program product can be received in apropagated signal, for example, over the transceiver 568 or the externalinterface 562.

The mobile computing device 550 may communicate wirelessly through thecommunication interface 566, which may include digital signal processingcircuitry where necessary. The communication interface 566 may providefor communications under various modes or protocols, such as GSM voicecalls (Global System for Mobile communications), SMS (Short MessageService), EMS (Enhanced Messaging Service), or MMS messaging (MultimediaMessaging Service), CDMA (code division multiple access), TDMA (timedivision multiple access), PDC (Personal Digital Cellular), WCDMA(Wideband Code Division Multiple Access), CDMA2000, or GPRS (GeneralPacket Radio Service), among others. Such communication may occur, forexample, through the transceiver 568 using a radio-frequency. Inaddition, short-range communication may occur, such as using aBluetooth, WiFi, or other such transceiver (not shown). In addition, aGPS (Global Positioning System) receiver module 570 may provideadditional navigation- and location-related wireless data to the mobilecomputing device 550, which may be used as appropriate by applicationsrunning on the mobile computing device 550.

The mobile computing device 550 may also communicate audibly using anaudio codec 560, which may receive spoken information from a user andconvert it to usable digital information. The audio codec 560 maylikewise generate audible sound for a user, such as through a speaker,e.g., in a handset of the mobile computing device 550. Such sound mayinclude sound from voice telephone calls, may include recorded sound(e.g., voice messages, music files, etc.) and may also include soundgenerated by applications operating on the mobile computing device 550.

The mobile computing device 550 may be implemented in a number ofdifferent forms, as shown in the figure. For example, it may beimplemented as a cellular telephone 580. It may also be implemented aspart of a smart-phone 582, personal digital assistant, or other similarmobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term machine-readable signal refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Although various implementations have been described in detail above,other modifications are possible. In addition, the logic flows depictedin the figures do not require the particular order shown, or sequentialorder, to achieve desirable results. In addition, other steps may beprovided, or steps may be eliminated, from the described flows, andother components may be added to, or removed from, the describedsystems. Accordingly, other implementations are within the scope of thefollowing claims.

1. (canceled)
 2. A computer-implemented method, comprising: receiving,by a computing device at a first time, a first voice query thatidentifies a physical object and a location of the physical object;transmitting the first voice query from the computing device to a remotecomputing system to cause the remote computing system to store anindication that the physical object is located at the location;receiving, by the computing device, at a second time that is afterhaving caused the remote computing system to store the indication thatthe physical object is located at the location, a second voice querythat identifies an attribute of the physical object, wherein theattribute of the physical object was not identified in the first voicequery; transmitting the second voice query to the remote computingsystem; receiving, by the computing device as a response to the secondvoice query, information that identifies the location of the physicalobject; and providing the information that identifies the location ofthe physical object for presentation to a user.
 3. Thecomputer-implemented method of claim 2, wherein the second voice queryincludes one or more terms not included in the first voice query thatidentifies the physical object.
 4. The computer-implemented method ofclaim 2, further comprising determining that the computing device islocated within a threshold distance of the location of the physicalobject, and in response, causing the computing device to presentinformation indicating the physical object is located in proximity ofthe computing device.
 5. The computer-implemented method of claim 2, inresponse to transmitting the first voice query from the computing deviceto the remote computing system to cause the remote computing system tostore the indication that the physical object is located at thelocation, causing the remote computing system to: generate in a datastructure a first node that represents the physical object; generate inthe data structure a second node that represents the location of thephysical object; and link the first node and the second node in the datastructure to represent the indication that the physical object islocated at the location.
 6. The computer-implemented method of claim 2,wherein receiving, by the computing device as the response to the secondvoice query, the information that identifies the location of thephysical object further comprises: receiving the information thatidentifies the location of the physical object in response to the remotecomputing system matching the attribute of the physical object in thesecond voice query to one or more attributes that were identified by theremote computing system but not identified in the first voice query. 7.The computer-implemented method of claim 5, further comprising: causingthe remote computing system to store data for the physical object in thedata structure that corresponds to a particular account for the user,wherein the data: (i) includes representations of one or more physicalobjects and one or more corresponding locations of the one or morephysical objects; and (ii) maps relationships among particular ones ofthe one or more physical objects and the one or more correspondinglocations of the one or more physical objects.
 8. Thecomputer-implemented method of claim 2, wherein the first voice querydoes not include information that identifies the location of thephysical object, and wherein the location of the physical object isdetermined based on a current location of the computing device at whichthe first voice query was originally spoken.
 9. The computer-implementedmethod of claim 2, wherein the second voice query that identifies anattribute of the physical object further comprises a retrieval commandthat requests for the location of the physical object identified in thefirst voice query.
 10. A system comprising: one or more computers andone or more storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform operations comprising: receiving, by a computingdevice at a first time, a first voice query that identifies a physicalobject and a location of the physical object; transmitting the firstvoice query from the computing device to a remote computing system tocause the remote computing system to store an indication that thephysical object is located at the location; receiving, by the computingdevice, at a second time that is after having caused the remotecomputing system to store the indication that the physical object islocated at the location, a second voice query that identifies anattribute of the physical object, wherein the attribute of the physicalobject was not identified in the first voice query; transmitting thesecond voice query to the remote computing system; receiving, by thecomputing device as a response to the second voice query, informationthat identifies the location of the physical object; and providing theinformation that identifies the location of the physical object forpresentation to a user.
 11. The system of claim 10, wherein the secondvoice query includes one or more terms not included in the first voicequery that identifies the physical object.
 12. The system of claim 10,further comprising determining that the computing device is locatedwithin a threshold distance of the location of the physical object, andin response, causing the computing device to present informationindicating the physical object is located in proximity of the computingdevice.
 13. The system of claim 10, in response to transmitting thefirst voice query from the computing device to the remote computingsystem to cause the remote computing system to store the indication thatthe physical object is located at the location, causing the remotecomputing system to: generate in a data structure a first node thatrepresents the physical object; generate in the data structure a secondnode that represents the location of the physical object; and link thefirst node and the second node in the data structure to represent theindication that the physical object is located at the location.
 14. Thesystem of claim 10, wherein receiving, by the computing device as theresponse to the second voice query, the information that identifies thelocation of the physical object further comprises: receiving theinformation that identifies the location of the physical object inresponse to the remote computing system matching the attribute of thephysical object in the second voice query to one or more attributes thatwere identified by the remote computing system but not identified in thefirst voice query.
 15. The system of claim 13, further comprising:causing the remote computing system to store data for the physicalobject in the data structure that corresponds to a particular accountfor the user, wherein the data: (i) includes representations of one ormore physical objects and one or more corresponding locations of the oneor more physical objects; and (ii) maps relationships among particularones of the one or more physical objects and the one or morecorresponding locations of the one or more physical objects.
 16. Thesystem of claim 10, wherein the first voice query does not includeinformation that identifies the location of the physical object, andwherein the location of the physical object is determined based on acurrent location of the computing device at which the first voice querywas originally spoken.
 17. The system of claim 10, wherein the secondvoice query that identifies an attribute of the physical object furthercomprises a retrieval command that requests for the location of thephysical object identified in the first voice query.
 18. Anon-transitory computer-readable medium storing software comprisinginstructions executable by one or more computers which, upon suchexecution, cause the one or more computers to perform operationscomprising: receiving, by a computing device at a first time, a firstvoice query that identifies a physical object and a location of thephysical object; transmitting the first voice query from the computingdevice to a remote computing system to cause the remote computing systemto store an indication that the physical object is located at thelocation; receiving, by the computing device, at a second time that isafter having caused the remote computing system to store the indicationthat the physical object is located at the location, a second voicequery that identifies an attribute of the physical object, wherein theattribute of the physical object was not identified in the first voicequery; transmitting the second voice query to the remote computingsystem; receiving, by the computing device as a response to the secondvoice query, information that identifies the location of the physicalobject; and providing the information that identifies the location ofthe physical object for presentation to a user.
 19. Thecomputer-readable medium of claim 18, wherein the second voice queryincludes one or more terms not included in the first voice query thatidentifies the physical object.
 20. The computer-readable medium ofclaim 18, further comprising determining that the computing device islocated within a threshold distance of the location of the physicalobject, and in response, causing the computing device to presentinformation indicating the physical object is located in proximity ofthe computing device.
 21. The computer-readable medium of claim 18, inresponse to transmitting the first voice query from the computing deviceto the remote computing system to cause the remote computing system tostore the indication that the physical object is located at thelocation, causing the remote computing system to: generate in a datastructure a first node that represents the physical object; generate inthe data structure a second node that represents the location of thephysical object; and link the first node and the second node in the datastructure to represent the indication that the physical object islocated at the location.